[RFC] Broaden support for profiling generated code#341
Conversation
|
The code changes seems reasonable, but I would like to know more about the motivation. What sort of non-ipython system would uses the |
|
I can think of a few precedents for generated code:
Perhaps the more salient point is that, IMO, it is not safe to assume that |
|
This makes some sense, but it would help to see an explicit example (i.e. a new test) that demonstrates how line-profiler is missing this case and how this patch fixes it. I don't understand how dataclasses could be used in a way that would cause trouble. I'd be interested in seeing that example. We are already using |
|
The specific use case I had in mind was this code generator here: https://github.com/inducer/pytools/blob/7638eacdb2ef690504bade0afc63fc11330b5e85/pytools/py_codegen.py. |
|
Is it possible to make a MWE that we can use as a test? I don't see an example of how to use |
|
from pytools.py_codegen import PythonFunctionGenerator
import linecache
cg = PythonFunctionGenerator("test_fn", args=(),
decorators=["from line_profiler import profile", "@profile"])
cg("return 42")
fn = cg.get_function()
# Mimic https://github.com/inducer/pytools/pull/295
linecache.cache["<generated code for 'test_fn'>"] = (None, None, ["from line_profiler import profile", "@profile", "def test_fn()", " return 42"], None)
fn()Without this PR (#341): $ export LINE_PROFILE=1
$ python t.py
Timer unit: 1e-09 s
0.00 seconds - <generated code for 'test_fn'>:2 - test_fn
Wrote profile results to profile_output.txt
Wrote profile results to profile_output_2025-05-16T171619.txt
Wrote profile results to profile_output.lprof
To view details run:
python -m line_profiler -rtmz profile_output.lprof
$ cat profile_output.txt
Timer unit: 1e-09 s
Total time: 2e-06 s
Could not find file <generated code for 'test_fn'>
Are you sure you are running this program from the same directory
that you ran the profiler from?
Continuing without the function's contents.
Line # Hits Time Per Hit % Time Line Contents
==============================================================
2
3
4 1 2000.0 2000.0 100.0
0.00 seconds - <generated code for 'test_fn'>:2 - test_fnWith this PR (#341): $ export LINE_PROFILE=1
$ python t.py
Timer unit: 1e-09 s
0.00 seconds - <generated code for 'test_fn'>:2 - test_fn
Wrote profile results to profile_output.txt
Wrote profile results to profile_output_2025-05-16T171545.txt
Wrote profile results to profile_output.lprof
To view details run:
python -m line_profiler -rtmz profile_output.lprof
$ cat profile_output.txt
Timer unit: 1e-09 s
Total time: 2e-06 s
File: <generated code for 'test_fn'>
Function: test_fn at line 2
Line # Hits Time Per Hit % Time Line Contents
==============================================================
2 @profile
3 def test_fn()
4 1 2000.0 2000.0 100.0 return 42
0.00 seconds - <generated code for 'test_fn'>:2 - test_fn |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #341 +/- ##
=======================================
Coverage 63.78% 63.78%
=======================================
Files 13 13
Lines 1041 1041
Branches 228 228
=======================================
Hits 664 664
Misses 316 316
Partials 61 61
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
|
Perfect, that's enough for me to at least accept the PR. It would be good you could do an additional PR where you add this as a test. Otherwise, I'll get to it eventually. It would involve putting that snippet in a some test/*.py file and adding pytools as an optional dependency to As an aside, I was browsing pytools and was amused to see many of the same ideas in ubelt and other stdlib extension libraries (e.g. boltons, strif, @jaraco 's tools, toolz, etc..) . I'm interested in exploring the question if there is a subset that should be added to the stdlib. |
Relevant PR (pyutils/line_profiler#341) has been merged
Relevant PR (pyutils/line_profiler#341) has been merged
use linecache directly only adjust filename if line_profiler loaded improve args fix maintain compatibility with pudb broaden line_profiler usage check more type annotations pyright ignores rename generated functions to avoid linecache warnings remove special pudb handling make sure names are unique fix bpr expand to support PythonCodeGenerator, refactor fixes remove special line_profiler handling Relevant PR (pyutils/line_profiler#341) has been merged Rework for linecache compatibility, improve tests Co-authored-by: Andreas Kloeckner <inform@tiker.net>
use linecache directly only adjust filename if line_profiler loaded improve args fix maintain compatibility with pudb broaden line_profiler usage check more type annotations pyright ignores rename generated functions to avoid linecache warnings remove special pudb handling make sure names are unique fix bpr expand to support PythonCodeGenerator, refactor fixes remove special line_profiler handling Relevant PR (pyutils/line_profiler#341) has been merged Rework for linecache compatibility, improve tests Co-authored-by: Andreas Kloeckner <inform@tiker.net>
Currently, packages that generate Python code aren't easy to profile with the line_profiler, unless doing some contortions to adjust the file name to something that triggers line_profiler to look into the linecache instead of the file system. This PR slightly broadens the existing support for ipython to support more code generators.